It is often desirable to use virtual input devices to input command and/or data into electronic systems, such as for example a computer system, a musical instrument, or a telephone. For example, although computers can now be implemented in almost pocket-size form factors, inputting data or commands on a mini-keyboard can be time consuming, awkward, and error prone. While many cellular telephones today can handle e-mail communication, actually inputting messages using their small touch pads can be difficult. A personal digital assistant (PDA) has much of the functionality of a computer but suffers from a tiny or non-existent keyboard.
Some interest has been shown to develop virtual interfaces for such small form-factor devices. A device with a virtual interface could determine when a user's fingers or stylus contacts a virtual keyboard, and what fingers contacted what virtual keys. The output of the system could perhaps be input to a device such as a PDA, in lieu of data that could otherwise be received by a mechanical keyboard. (The terms “finger” or “fingers”, and “stylus” are used interchangeably throughout this application.) In this example a virtual keyboard might be a piece of paper, perhaps that unfolds to the size of a keyboard, with keys printed thereon, to guide the user's hands. It is understood that the virtual keyboard or other input device is simply a work surface and has no sensors or mechanical or electronic components. The paper and keys would not actually input information, but the interface of the user's fingers with portions of the paper, or if not paper, portions of a work surface, whereon keys would be drawn, printed, or projected, could be used to input information to the PDA. A similar virtual device and system might be useful to input e-mail to a cellular telephone. A virtual piano-type keyboard might be used to play a real musical instrument.
Previously, camera-based systems have been proposed that detect or sense where the user's fingers are relative to a virtual device. These systems achieve their sensing goals in different ways.
U.S. Pat. No. 5,767,848 to Korth (1998) entitled “Method and Device for Optical Input of Commands or Data” attempts to implement virtual devices using a two-dimensional TV video camera. A video camera captures luminosity information, which by itself cannot determine the relative position of an object with respect to a table. Korth processes this luminosity data to (1) distinguish the hands from the background, (2) identify the fingertips of the hands, (3) determine the velocity of the fingers over several frames, and (4) identify when the velocity stops abruptly, in order to determine that the finger has hit the table. Korth would require computation of the contour of a user's fingers, finger position relative to the virtual device, and a determination of finger movement. This method may work in contrived environments, where the background behind the fingers is known, or at least known to have different luminosity from that of the fingers themselves. As a consequence, Korth's camera would have to be placed at some distance from the table, looking down on it. Otherwise, the camera would see the user's palms or shirt, or even the room or environment around the user. This would lead to unpredictable backgrounds, and finger/background separation would be problematic. A down-looking camera is essential for Korth's system also because of a different reason: In order to determine the position of contact of a finger with the plane, the projection rays of Korth's camera must intersect the plane itself, so that a one-to-one correspondence can be established between points on the plane and points in the image. For all these reasons, Korth's camera must be placed several centimeters away from the plane containing the keyboard's active area. A camera that is, say, a mere centimeter away from the plane would have to see the plane at an appreciable angle only for active areas that are a few centimeters away from the camera itself. This would result into very small keyboards, perhaps 3 by 6 centimeters in size.
U.S. Pat. No. 6,115,128 to Vann discloses methods that can capture up to 6 degrees of freedom of the position of an object in space. A solid has six degrees of freedom which are the (x,y,z) position of its center of gravity and the 3 angles that defines its orientation which are the yaw pitch and roll angles. Vann's method is applicable to rigid solids. An object consisting of two solid objects connected by a long string has more than six degrees of freedom. Objects that are composed of a number of solids and objects that are deformable have many more than 6 degrees of freedom. Indeed objects such as a single hand may have in excess of 30 degrees of freedom, not even accounting for the deformation of the fingers as they strike the table. Vann's method are therefore not applicable to hands and fingers.
In European patent number 99111323.4, Carau uses a sensor system that is little more than a stereo camera system that uses horizontal cameras to view fingers or other objects illuminated by a light beam. Carau's system will not provide information from which relevant position can be uniquely determined without ambiguity. Prior art attempts to use conventional two-dimensional cameras are known in the art, and suffer extremely acute occlusion problems in addition to ambiguity-creating problems. Satisfactory operation of such systems requires that each stereo sensor be able to view all user fingers in the region of interest. But in practice, one finger can very easily occlude the sensor-view of another user finger. By definition, stereo cameras such as found in a Carau-type system, are spaced-apart along one axis (e.g., along the left-to-right width of a virtual keyboard), and as a result, finger occlusion is a very real and constant problem. Also, since stereo-camera systems require that both sensors view the user's fingers, the likelihood of occlusion is essentially more than doubled in practice. The occlusion problems would make a stereo system impractical for a virtual input device application.
In U.S. patent application Ser. No. 09/187,903, Dunton describes an image projection system using LCD or DLP to project a keyboard pattern. Dunton suggests that a scanning sensor can be used to detect the movement of user finger. The application suggests that the invention will require-multiple-video cameras (up to three, since more cameras result in better three-dimensional imaging).